Dropout training for Hidden Unit CRFs

A very commonly faced issue while training prediction models using machine learning is overfitting. Dropout is a recently developed technique designed to counter this issue in deep neural networks and has also been extended to other algorithms like SVMs. In this project, we formulate and study the application of Dropout to Hidden Unit Conditional Random Fields (HUCRFs). HUCRFs use binary stochastic hidden units to model the underlying latent structure in the data and hence are similar to Neural Networks. The Dropout technique proposed in this report involves dropping of a random set of hidden states for each training example. For test inference, we formulate an inference equation that is equivalent to computing expected prediction using the ensemble of all Dropout models. In this report, we also evaluate our Dropout train and test formulations when used for OCR. We evaluate the models by varying complexity of the models and the amount of dropout. Results show that HUCRF models trained with dropout consistently perform better than baseline HUCRF.